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[Name of Document] SPECIHCATION 
[Title of the Invention] 

PORTABLE INFORMATION PROCESSING TERMINAL, INFORMATION 
INPUT/OUTPUT SYSTEM AND INFORMATION INPUT/OUTPUT METHOD 
[Claims] 
[Claim 1] 

A portable information processing terminal adapted to be connected to one or 
more than one targets by way of a network, said terminal comprising: 

an imaging means for imaging visible identification information possessed by 
the targets: 

an identification means for identifying the targets on the basis of the 
identification information imaged by said imaging means; and 

a connection means for establishing connection between itself and the targets 
identified by said identification means. 
[Claim 2] 

The portable information processing terminal according to claim 1, further 
comprising: 

an image confirmation means for confirming the images of the targets including 
said identification information. 
[Claim 3] 

An information input/output system comprising: 
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a user terminal to be used by a user; 

one or more than one targets connected to said user terminal in a format adapted 
to transfer of information and including visible identification information; 

an identification means for an identifying the targets on the basis of the 
identification information imaged by said imaging means; and 

a connection means for establishing connection between the user terminal and 
the targets identified by said identification means. 
[Claim 4] 

The information input/output system according to claim 3, further comprising: 
an image confirmation means for confirming the images of the targets including 
said identification information. 
[Claim 5] 

The information input/output system according to claim 3, wherein 
said imaging means is provided integrally in said user terminal. 
[Claim 6] 

The information input/output system according to claim 3, wherein 
said imaging means is connected to said user terminal by way of a cable. 
[Claim 7] 

The information input/output system according to claim 3, wherein 
said imaging means is connected to said user terminal by way of a network. 
[Claim 8] 
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The information input/output system according to claim 3, wherein 
said user terminal is connected to said targets by way of a network; 
said targets have respective network addresses; 

said information input/output system further comprising a data base means for 
controlling the correspondence between the identification information of each target 
and its network address; and 

said connection means being adapted to connect the user terminal and the 
targets by referring to said data base means for the network address of each target 
corresponding to the identification information. 
[Claim 9] 

The information input/output system according to claim 3, wherein said 
connection means sustains the connection between the targets and said user terminal 
as long as the imaging means is imaging the targets or their identification information. 
[Claim 10] 

The information input/output system according to claim 3, wherein 
said user terminal includes a display screen; and 

the images taken by said imaging means is displayed on said display screen. 
[Claim 11] 

The information input/output system according to claim 3, wherein 
said user terminal includes a display screen; 

the images taken by said imaging means is displayed on said display screen; 

3 



and said connection means sustains the connection between the targets and said user 
terminal as long as the targets identified by said identification means are displayed on 
said display screen. 
[Claim 12] 

The information input/output system according to claim 3, wherein 
said user terminal includes a display screen; 

the images taken by said imaging means is displayed on said display screen; 

and 

said images of the targets taken by said imaging means is held on said display 
screen as long as said connection means sustains the connection between the targets 
and said user terminal. 
[Claim 13] 

The information input/output system according to claim 3, wherein 
said user terminal has a storage means for storing the taken images of said targets 
identified by said identification means. 
[Claim 14] 

The information input/output system according to claim 3, wherein 

said user terminal has a storage means for storing the taken images of said 

targets identified by said identification means; and 

said connection means establishes connection between said user terminal and 

relevant taken targets in response to the display of said image on said display screen. 



[Claim 15] 

The information input/output system according to claim 3, wherein 

said user terminal has a storage means for storing the taken images of said 

targets identified by said identification means; and 

all the taken images retrieved from said storage means are displayed on the 

display screen and said the connection means establishes connection between said user 

terminal and relevant taken targets in response to the selection of said image from said 

displayed images. 
[Claim 16] 

The information input/output system according to claim 3, further comprising: 
an application means for transferring predetermined data and/or commands 
between the user terminal and the targets being connected. 
[Claim 17] 

The information input/output system according to claim 3, wherein 
said user has a display screen; and 

said application means transfers the page being displayed on the display screen 
to a projector, or a target, as an image to be projected as long as said connection 
means sustains the connection between the projector and the user terminal. 
[Claim 18] 

The information input/output system according to claim 3, wherein 
said user terminal has a keyboard or some other user input means; and 



said application means transfers the user input data input by using said user 
input means as user input data on a computer system, or a target, as long as said 
connection means sustains the connection between the computer system and the user 
terminal. 
[Claim 19] 

The information input/output system according to claim 3, wherein 
said user has a display screen; and 

said application means obtains the data to be shown from targets and display 
them on the display screen as long as said connection means sustains the connection 
between said targets including said data to be shown and updated regularly or 
irregularly and the user terminal. 
[Claim 20] 

The information input/output system according to claim 3, wherein 
said application means receives alarms from a target, or a device having a alarm 
feature and execute said alarm on said user terminal as long as said connection means 
sustains the connection between the target and the user terminal. 
[Claim 21] 

The information input/output system according to claim 3, wherein 
said application means transmits the video recording appointment information input 
at said user terminal to a target, or a video recording/reproduction apparatus having 
a video recording appointment feature so long as said connection means sustains the 
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connection between said video recording/reproduction apparatus and the user 
terminal. 
[Claim 22] 

The information input/output system according to claim 3, wherein 
said user has a display screen; and 

said application means displays an image of operation for controlling remotely 
controllable targets on the display screen and transmit the user operation displayed on 
the display screen to the targets as long as said connection means sustain the 
connection between the target and the user terminal. 
[Claim 23] 

An information input/output method for processing user inputs/outputs in an 
information space comprising a user terminal to be used by a user; 

one or more than one targets connected to said user terminal in a format adapted 
to transfer of information and including visible identification information, an 
identification means for identifying the targets on the basis of the identification 
information imaged by said imaging means and a connection means for establishing 
connection between the user terminal and the targets identified by said identification 
means; said method comprising: 

(a) step of identifying the targets on the basis of the identification information 
imaged by said imaging means; and 

(b) a step of establishing connection between the targets identified by said 
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identification means and said user terminal. 
[Claim 24] 

The information input/output method according to claim 23, wherein said 
information space is further provided with an image confirmation means for 
confirming the images of the targets including said identification information. 
[Claim 25] 

The information input/output method according to claim 23, wherein 
said user terminal and said targets are connected by a network; and 
said targets have respective network addresses; whereas 

said information space is provided with a data base means for controlling the 
correspondence between the identification information of each target and the network 
address; and 

connection to said targets is realized by referring to said data base means for 
the network address of each target corresponding to the identification information in 
said step (b) of establishing connection, 
[Claim 26] 

The information input/output method according to claim 23, wherein the 
connection between the targets and said user terminal is sustained as long as the 
imaging means is imaging the targets or their identification information in said step 
(b) of establishing the connection. 
[Claim 27] 
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The information input/output method according to claim 23, wherein 

said user terminal includes a display screen and the images taken by said 

imaging means is displayed on said display screen; and 

the connection between the targets and said user terminal is sustained as long 

as the targets identified by said identification means are displayed on a display screen 

in said step (b) of establishing the connection. 

[Claim 28] 

The information input/output method according to claim 23, wherein 

said user terminal includes a display screen and the images taken by said 

imaging means is displayed on said display screen; and 

said images of the targets are be held on a display screen as long as said 

connection means sustains the connection between the targets and said user terminal. 
[Claim 29] 

The information input/output method according to claim 23, wherein 

said user terminal has a display screen and a storage means for storing the taken 

images of said targets identified by said identification means; and 

said step (b) is executed to establish connection established between said user 

terminal and relevant taken targets in response to the display of said image on said 

display screen. 

[Claim 30] 

The information input/output method according to claim 23, wherein 
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said user terminal has a display screen and a storage means for storing the taken 
images of said targets identified by said identification means; and 

said method further comprises (c) a step of displaying all the taken images 
retrieved from said storage means on the display screen; and 

said step (b) is executed to establish connection between said user terminal and 
relevant taken targets in response to the selection of said image from said displayed 
images. 
[Claim 31] 

The information input/output method according to claim 23, further 
comprising: (d) an application step of transferring predetermined data and/or 
commands between the user terminal and the targets being connected. 
[Claim 32] 

The information input/output method according to claim 23, wherein 
said user terminal has a display screen; and 

the page being displayed on the display screen to a projector, or a target, is 
transferred as an image to be projected as long as the connection between the projector 
and the user terminal is sustained in said application step (d). 
[Claim 33] 

The information input/output method according to claim 23, wherein 

said user terminal has a keyboard or some other user input means; and 

the user input data input by using said user input means can be transferred as 
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user input data on a computer system, or a target, as long as the connection between 
the computer system and the user terminal is sustained in said application step (d). 
[Claim 34] 

The information input/output method according to claim 23, wherein 
said user terminal has a display screen; and 

data to be shown can be obtained from the target and displayed on the display 
screen as long as the connection between said targets including said data to be shown 
and updated regularly or irregularly and the user terminal is sustained in application 
step (d). 
[Claim 35] 

The information input/output method according to claim 23, wherein the alarms 
are received from a target and executed on said user terminal as long as the connection 
between a device having an alarm feature, or a target, and the user terminal is 
sustained in said application step (d). 
[Claim 36] 

The information input/output method according to claim 23, wherein the video 
recording appointment information input at said user terminal to a video 
recording/reproduction apparatus having a video recording appointment feature, or a 
target, is transmitted to the target so long as the connection between said video 
recording/reproduction apparatus and the user terminal is sustained in said application 
step (d). 
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[Claim 37] 

The information input/output method according to claim 23, wherein 
said user terminal has a display screen; and 

an image of operation for controlling remotely controllable targets is displayed 
on the display screen and the user operation displayed on the display screen can be 
transmitted to the targets as long as said the connection between the target and the user 
terminal is sustained in said application step (d). 
[Detailed Description of the Invention] 
[0001] 

[Technical Field of the Invention] 

This invention relates to a proposal for a user interface environment in terms 
of operations of issuing commands to a computer and, more particularly, it relates to 
a proposal for a user interface environment that allows the user to issue commands in 
an intuitive and easily understandable manner to a computer. 
[0002] 

Still more particularly, the present invention relates to a proposal for a user 
interface environment that allows the user to operate various devices in an information 
space in an integrated, intuitive and easily understandable manner on user's computer. 
In such a user interface environment, the user can operates various devices in an 
information space in an intuitive and easily understandable manner on a computer 
display. 
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[0003] 
[Prior Art] 

As a result of technological innovations in recent years workstations and 
personal computers, which are compact general purpose computer systems with high 
added values and enhanced functional features have been developed and marketed at 
low prices. Such computer systems are almost ubiquitous in various research 
institutions including colleges and universities, commercial firms and offices and 
ordinary homes where people spend their daily lives. 
[0004] 

One of the recent trends in the field of computer technology is down-sizing of 
computers. Particularly, compact and lightweight portable computers (mobile 
computers) have become very popular owing to the recent development of 
semiconductor technology that has made very small LSI (large scale integration) chips 
commercially available and boosted the processing capacity of computer processors. 
[0005] 

Portable computers include so-called "notebook PCs" and palmH:op sized 
computers, which are also referred to as "PDA (personal digital assistants)". These 
and other computers can be operated with systems in an mobile environment so that 
the user can use his or her own computer almost anywhere, indoors, outdoors, in 
moving vehicles, etc. 
[0006] 
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Most computer systems provide the user with an interactive processing 
environment so that the computer system operates in response to the command issued 
to it by the user and the outcome of the processing operation is displayed on the 
display screen of the system. Currently, there is occurring a shift from the character- 
based user input environment or "the GUI (character user interface) environment" 
such as DOS (disk operating system) shell screen to the graphic-based user input 
environment or "the GUI (graphic user interface) environment". In the GUI 
environment, a disk top unit simulating a computer system and numerous icons are 
provided for the display screen. 
[0007] 

In the case of a disk top type computer provided with a GUI, all resource 
objects to be handled in the computer system including files are expressed by icons. 
The user can intuitively issue commands by directly operating icons on the display 
screen that represent respectively programs, data, holders and devices by means of a 
mouse and other I/O devices (with motions such as click, drag and drop, etc.). 
Therefore, the user is no longer required to memorize specific commands and learn 
how to handle them. Nor he or she is required to learn how to operate the keys of the 
keyboard for issuing commands, which is normally a very cumbersome task. 
[0008] 

Another of the recent trends in the field of computer technology is 
interconnection. Computers may be interconnected typically by way of local 
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interfaces such as serial ports and parallel ports or by way of network interface cards 
(NICs) so that they may be connected to each other by way of one or more than one 
telecommunication networks. From the viewpoint of interconnection of computers, 
telecommunication networks include LANs (local area networks) provided on single 
sites, WANs (wide area networks) realized by connecting LANs by way of dedicated 
lines and the internet which is a global telecommunication network realized by 
connecting various networks. 
[0009] 

Interconnection of computers provides advantages including that information 
and computer resources are shared by a number of computer users, that information 
is delivered and disseminated highly efficiently and that a plurality of computers can 
be driven in a cooperative and concerted manner to carry out a common mission. For 
instance, an operation of processing a digital object can be carried out highly 
efficiently when a plurality of computers are made to cooperate in an intensive way. 
[0010] 

Particularly, in recent years, the user can link his or her computer easily to a 
telecommunication network as a result of the advent of proximity telecommunication 
systems such as wireless LAN that is a wireless version of Ethernet and blue tooth 
The computers and peripheral devices (e.g., printers) installed in a building can mostly 
be connected to a teleconmiunication network so that they may be shared for operation 
by a plurality of users. Additionally, computer users can carry their own computers 
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to any place they want and link them to a telecommunication network without 

difficulty. 

[0011] 

In a distributed computing environment, the computers connected to a network 
are in a "transparent" state and they can be operated by means of an application 
program regardless if it is stored in a local storage disk or in a remote disk. In other 
words, the user is no long required to know the locations of resource objects such as 
programs and data because they can be exchanged seamlessly among computers. 
[0012] 

The user can link his or her computer to a telecommunication network without 
any difficulty. However, if the user wants to specify a particular computer or 
peripheral device (target) to which data are to be transferred by way of the network, 
he or she has to know its ID (or its address) if it is located just in front of the user. In 
other words, even in a transparent and distributed computing environment, user 
operations are rather indirect and the related computers are not cooperative so that the 
user cannot act intuitively. 
[0013] 

For example, if the user want to transfer an object between two or more than 
two computers in a currently available GUI environment, he or she is required to 
display a list of the devices linked to the network on an explorer screen and find out 
the icons simulating the respective computers before proceeding to a "drag and drop" 
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operation. 
[0014] 

A technique for solving the user operation problem may be the use of a real 
world-oriented user interface, which represents the technology for expanding the 
scope of the user operation from the "digital space" produced on a display screen of 
a computer to the space of the real world in an discontinued way. With such a real 
world-oriented interface, the display screen of the computer may be expanded to the 
surface of the disk on which the computer is placed and also to the surface of the walls 
of the room where the computer is located. In other words, the information space of 
the user expands beyond the limit of the digital space of the stand-alone computer to 
the entire room so that digital objects of the digital space of the computer and physical 
objects of the real world can be handled in an integrated manner. 
[0015] 

For instance, "pick-and-drop^^^" is an expansion of user operation "drag-and 
drop" that is reliably being utilized in the GUI environment and represents the 
technology of advanced interactions for mapping the transfer of objects such as data 
among a plurality of computers into an intuitive motion in the real world. 
[0016] 

With the "pick-and-drop" technique, seamless "pick-and-drop" operations can 
be realized as a result of cooperation of a plurality of disk top computers. For 
example, when an object is dragged by operating the user input device that may be a 
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track ball or a stick of a disk top computer to move the pointer in the display screen 
beyond an edge thereof, the drag operation may be succeeded to the display screen of 
the adjacently located disk top computer so that the pointer of the latter display screen 
appears, dragging the object. 
[0017] 

On the other hand, the pointer in the display screen of the former disk top 
computer disappears, while dragging the object. This operation of the pointer of the 
former disk top computer picking up an object is referred to as "pick" while the 
operation of causing the pointer of the latter disk top computer to appear in order to 
continuously drag the object is referred to as "drop". Thus, the entity of the object 
(e.g., a data file) is transferred behind the screen between the two computers by way 
of the network. In other words, the succeeded drag operation accompanies a 
movement of the entity of the object. FIG. 21 schematically illustrate an "pick-and- 
drop" operation of an digital object of adjacently located computers. 
[0018] 

As for "pick-and-drop" operations, refer to Japanese Patent Application Laid- 
Open Publication No. 1 1-53236 whose assignee is the applicant of the present patent 
application, specifically passages [0052] through [0059] and FIGS. 6 through 9 of the 
above patent document. 
[0019] 

On the other hand, with "media blocks^^^", the user can assign data such as 
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images to blocks having respective IDs and move the data by carrying the 
corresponding blocks or rearrange the data by physically rearranging the latter. 
Therefore, blocks may be arranged in a dedicated slot or on a dedicated rail of a 
computer and data may be moved or rearranged by transferring corresponding blocks 
in the computer. 
[0020] 

"Hyper-drag^"*^" provides a user interface environment for the portable user 
computer to cooperate v^ith a shared computer designed to use a desk top or a wall as 
display screen. For example, "hyper-drag" operations can be supported by 
introducing a camera-based object recognition system into the information 
environment. More specifically, the image of a computer display screen may be 
projected onto the surface of a disk top or a v^all by means of a projector. 
Furthermore, both a projected image that is an object in the logical space of the 
computer and an object of the real world (that may be found on a disk top) may be 
recognized on the basis of a corresponding image picked-up by a camera and the 
motions of the objects may be traced so that the user can operate the computer 
interactively on the display screen that is now expanded to the disk top or the wall. 
[0021] 

With "hyper-drag", the digital space of a computer is expanded into the space 
of the real world. For example, a drag operation conducted on a user computer is 
made to continue on the plane where the computer is placed (e.g., the surface of a 
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disk) (see FIG. 22) and the plane or the space containing it may be expanded to an 
wall of the workshop (see FIG. 23). In other words, the space of the workshop where 
the computer is placed can be used as if it were the disk top input device of the 
computer to operate for the object. 
[0022] 

As for "hyper-drag" operations, refer to Japanese Patent Application No. 11- 
316461, the assignee of which is the applicant of the present patent application 
[0023] 

The above described known techniques make it possible to expand the digital 
space of a computer into the space of the real world so that operations for handling 
objects can be carried out easily and intuitively by making a plurality of computers 
cooperate, 
[0024] 

However, the above known techniques are adapted to transfer data only once 
or only a block of data and necessary connections are established only through user 
operations. Nor any sustained connections are assumed for those techniques. In other 
words, the user is required to handle objects in a digital space or in the real space for 
each transfer of data and, once the object handling or the data transfer is over, the 
computer is disconnected from the related devices. 
[0025] 

However, the user may retain interest in any of the devices after transmitting 
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data and/or one or more than one commands to an external device. For example, he 
or she may want to monitor the operation of a device after issuing a command to it or 
add a command or operation for handling the device. Then, since the connections of 
the device are sequentially cut, the user is required to operates the computer almost 
incessantly in order to allow the it to continuously communicate with the device. In 
other words, it is not possible to keep the connection in a standby state until the device 
is brought into a predetermined certain state. 
[0026] 

For example, while it is possible to specify an object of connection typically by 
using the technique of "pick-and-drop", neither the computer nor the object is 
provided with means for showing the state of "being connected" or what it is 
connected to so that the object of connection should be referred to only by using its 
name except the moment when it is connected to the computer. In other words, it is 
necessary that the state of connection is visible to the user in order to establish a 
sustained connection by means of a physical operation like "pick-and-drop". 
[0027] 

With "pick-and-drop", the stylus of a digitizer is used for picking up a 
displayed object from a display screen and drop it in another display screen. The 
technique of "pick-and-drop" resembles an operation of picking up a physical object 
and drop it. The stylus cannot be operated to handle an object that is out of the reach 
of the user. In other words, a remote information device cannot be picked up and 
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dropped by remote control. 
[0028] 

With the technique of "hyper-drag", on the other hand, it is possible to expand 
an information space from the digital space of a computer to an operational space of 
the real world to realize a spatial continuity for the handling of information. As a 
result, a digital object and a physical object can be handled in a cooperative manner. 
More specifically, a digital object can be dragged and dropped onto a physical object 
to establish a new link between the physical object and the digital object and a digital 
object can be attached to a physical object and moved in the real space. Furthermore, 
a digital object can be drawn out of a physical object. 
[0029] 

However, while an informational environment can be made to three- 
dimensionally continue in the real world with the technique of "hyper-drag", the 
technique is not adapted to handle a physical object on a computer display screen 
digitally or by means of a command. For example, while a digital object may be 
attached or transferred to a physical object such as a video player, the recording or 
replay operation of the video player is not something that can be handled by a 
command issued by way of a computer. 
[0030] 

In addition to the studies as cited above, a number of other studies have been 
reported in the field of real world-oriented interface. For example, there are known 
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studies on objects and places that are provided with a physical (and hence visible) 
identifier so that information may be displayed or transferred by means of computer 
resources. Systems as proposed by Chameleon NaviCam CyberGuide 
UbiquitousLinks and document [9] may be regarded as interfaces through which 
a block of data can be taken out from an object or a place. 
[0031] 

Chameleon is a system for displaying information by means of a magnetic 
sensor depending on the position and the posture of a hand-held display and 
commands can be input by moving the display with the system. 
[0032] 

NaviCam is a system adapted to use an ID that can be identified by image 
recognition to provide information according to the identified ID. Navicom can be 
combined with gyros and voice recognition. 
[0033] 

CyberGuide is a system adapted to display guide information on a PDA 
according to the position of the user by means of an ID using ultraviolet rays. The 
data to be displayed is stored in the PDA in advance. 
[0034] 

UbiquistousLinks is a system adapted to establish a link from an object or a 
place to a resource object on a WWW (world wide web) by using an ID tag or an 
ultraviolet ray ID. With such a system, data can be transferred and the linked 
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destination can be shifted sequentially in the order of the identified Ids. 
[0035] 

The document [9] proposes a system resembling UbiquitousLinks that can 
establish a link from an object to an electronic information source such as WWW by 
attaching a tag to the object. 
[0036] 

Additionally, the systems proposed by "Infospuit" ^^^\ "Extended Disk Top 
Environment ^^'^ and document [12] are adapted to transfer or assign data to an object 
in addition to drawing out data from an object. They are also adapted to bidirectional 
data transfer. 
[0037] 

Infospuit is a system where an object in the real world and a computer can 
exchange information by way of a device that can recognize two-dimensional codes 
such as a remote control unit. It is also provided with a feature of switching its 
operation depending on the destination of data transfer. 
[0038] 

Extended Disk Top Environment is a system adapted to virtually arrange icons 
in space by recognizing the images picked up by a camera. With the system, any 
printer in the picked up image can be registered in advance and a printing operation 
can be carried out by dropping the icon. 
[0039] 
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Document [12] proposes a system adapted to recognize a place or an object by 
means of a two^imensional matrix or an infrared ray ID and see a picture or a memo 
showing voiced information and paste it to the place or the object. 
[0040] 

However, none of the above described real world-oriented interfaces assume 
sustained connections. 
[0041] 

Meanwhile, the network environment for interconnecting computers have been 
remarkably improved in recent years and nowadays users are not required to pay 
particular efforts for connecting computers to a telecommunication network 
(previously described). Furthermore, many devices designed for use in ordinary daily 
lives are currently provided with information processing means to make themselves 
compatible with a telecommunication network. For example, video recorders, 
television sets, refrigerators and electric kettles are information-oriented and can be 
remotely controlled by means of computer and a network. 
[0042] 

However, when the user tries to specify a particular electric home appliance on 
the network, he or she is required to know its ID (or its address) and how to issue a 
command to it. In other words, the operation of issuing a command by way of a 
network is an indirect one and cannot be conducted in an intuitive way. 
[0043] 
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Electric home appliances are provided with excellent user interfaces such as a 
control panel/button and ordinary users who do not know how to operate computers 
can issue commands to them and use them almost without learning anything about 
electronic control systems. However, remote control systems involving networks are 
not as convenient as those of electric home appliances. 
[0044] 

[Problems to be Solved by the Invention] 

In view of the above circumstances, it is therefore an object of the present 
invention to provide a comfortable user interface environment where users can issue 
commands to computers in an easy and intuitive way. 
[0045] 

Another object of the present invention is to provide a comfortable user 
interface environment where users can operate various devices arranged in an 
information space in an easy, intuitive and integrated way by means of the user 
computer. 
[0046] 

Still another object of the present invention is to provide a comfortable user 
interface environment where users can control and operate various devices arranged 
in an information space in an easy and intuitive way on the user computer display. 
[0047] 

Still another object of the present invention is to provide a comfortable user 
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interface environment where users can sustainedly connect various devices arranged 

in an information space to computers. 

[0048] 

A further object of the present invention is to provide a comfortable user 
interface environment where users can sustainedly connect various devices arranged 
in an information space to computers and confirm the connected state on the user 
computer display. 
[0049] 

A still further object of the present invention is to provide a comfortable user 
interface environment where users can sustainedly connect various devices arranged 
in an information space to computers and cause them to exchange commands and data. 
[0050] 

[Means to Solve the Problem] 

In the first aspect of the present invention, the above objects and other objects 
are achieved by providing a portable information processing terminal adapted to be 
connected to one or more than one targets by way of a network, said terminal 
comprising: 

an imaging means for imaging visible identification information possessed by 
the targets: 

an identification means for identifying the targets on the basis of the 
identification information imaged by said imaging means; and 
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a connection means for establishing connection between itself and the targets 
identified by said identification means. 
[0051] 

The portable information processing terminal in the first aspect of the invention 
may further comprise an image confirmation means for confirming the images of the 
targets including said identification information. 
[0052] 

In the second aspect of the present invention, there is provided an information 
input/output system comprising: 

a user terminal to be used by a user; 

one or more than one targets connected to said user terminal in a format adapted 
to transfer of information and including visible identification information; 

an imaging means for imaging visible identification information; 

an identification means for an identifying the targets on the basis of the 
identification information imaged by said imaging means; and 

a connection means for establishing connection between the user terminal and 
the targets identified by said identification means. 
[0053] 

The information input/output system in the second aspect of the invention may 
further comprise an image confirmation means for confirming the images of the 
targets including said identification information. 
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[0054] 

In the information input/output system in the second aspect of the invention, 
said imaging means may be provided integrally in said user terminal or connected to 
said user terminal by way of a cable or a network. 
[0055] 

In the information input/output system in the second aspect of the invention, 
it may be so arranged that said user terminal is connected to said targets by way of a 
network; and 

said targets have respective network addresses; 

said information input/output system further comprising a data base means for 
controlling the correspondence between the identification information of each target 
and its network address. With this arrangement, said connection means can connect 
the user terminal and the targets by referring to said data base means for the network 
address of each target that corresponds to the identification information. 
[0056] 

In the information input/output system in the second aspect of the invention, 
said connection means may sustain the connection between the targets and said user 
terminal as long as the imaging means is imaging the targets or their identification 
information. 
[0057] 

Said user terminal may include a display screen; and 
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the images taken by said imaging means may be displayed on said display 
screen. With this arrangement, said connection means may sustain the connection 
between the targets and said user terminal as long as the targets identified by said 
identification means are displayed on said display screen. Alternatively, said images 
of the targets taken by said imaging means may be held on said display screen as long 
as said connection means sustains the connection between the targets and said user 
terminal. 
[0058] 

Said user terminal may have a storage means for storing the taken images of 
said targets identified by said identification means. With this arrangement, said 
connection means may establish connection between said user terminal and relevant 
taken targets in response to the display of said image on said display screen. 
Alternatively, it may be so arranged that all the taken images retrieved from said 
storage means are displayed on the display screen and the connection means 
establishes connection between said user terminal and relevant taken targets in 
response to the selection of said image from said displayed images. 
[0059] 

The information input/output system in the second aspect of the invention may 
further comprise an application means for transferring predetermined data and/or 
commands between the user terminal and the targets being connected. 
[0060] 
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Said application means can transfer the page being displayed on the display 
screen to a projector, or a target, as an image to be projected as long as said 
connection means sustains the connection between the projector and the user terminal. 
[0061] 

If the user terminal comprises a keyboard or some other user input means, said 
application means can transfer the user input data input by using said user input means 
as user input data on a computer system, or a target, as long as said connection means 
sustains the connection between the computer system and the user terminal. 
[0062] 

Said application means can obtain the data to be shown from targets and display 
them on the display screen as long as said connection means sustains the connection 
between said targets including said data to be shown and updated regularly or 
irregularly and the user terminal. 
[0063] 

Said application means can receive alarms from a target and execute said alarm 
on said user terminal as long as said connection means sustains the connection 
between a device having an alarm feature, or a target, and the user terminal. 
[0064] 

Said application means can transmit the video recording appointment 
information input at said user terminal to a video recording/reproduction apparatus 
having a video recording appointment feature, or a target, so long as said connection 
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means sustains the connection between said video recording/reproduction apparatus 

and the user terminal. 

[0065] 

Said application means can display an image of operation for controlling 
remotely controllable targets on the display screen and transmit the user operation 
displayed on the display screen to the targets as long as said connection means sustain 
the connection between the target and the user terminal. 
[0066] 

In the third aspect of the invention, there is provided an information 
input/output method for processing user inputs/outputs in an information space 
comprising a user terminal to be used by a user; 

one or more than one targets connected to said user terminal in a format adapted 
to transfer of information and including visible identification information, an 
identification means for identifying the targets on the basis of the identification 
information imaged by said imaging means and a connection means for establishing 
connection between the user terminal and the targets identified by said identification 
means; said method comprising: 

(a) step of identifying the targets on the basis of the identification information 
imaged by said imaging means; and 

(b) a step of establishing connection between the targets identified by said 
identification means and said user terminal. 
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[0067] 

With the above arrangement, it is possible to connect said user terminal and 
said targets to a network and assign specific network addresses to the respective 
targets. A data base means for controlling the correspondence between the 
identification information of each target and its network address may be provided in 
said information space. If such is the case, the user terminal and the targets can be 
connected by referring to said data base means for the network address of each target 
that corresponds to the identification information in said step (b) of establishing the 
connection. 
[0068] 

Said information space may be further provided with an image confirmation 
means for confirming the images of the targets including said identification 
information. 
[0069] 

It may be so arranged that the connection between the targets and said user 
terminal is sustained as long as the imaging means is imaging the targets or their 
identification information in said step (b) of establishing the connection. 
[0070] 

It may also be so arranged that the connection between the targets and said user 
terminal is sustained as long as the targets identified by said identification means are 
displayed on a display screen in said step (b) of estabUshing the connection. 
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Alternatively, it may be so arranged that the said images of the targets are be held on 
a display screen as long as said connection means sustains the connection between the 
targets and said user terminal. 
[0071] 

Said user terminal may have a storage means for storing the taken images of 
said targets identified by said identification means. With this arrangement, said step 
(b) is executed to establish connection established between said user terminal and 
relevant taken targets in response to the display of said image on said display screen. 
[0072] 

It may be so arranged that said method further comprises (c) a step of 
displaying all the taken images retrieved from said storage means on the display 
screen; and 

said step (b) is executed to establish connection between said user terminal and 
relevant taken targets in response to the selection of said image from said displayed 
images. 
[0073] 

It may be so arranged that said method further comprises (d) an application step 
of transferring predetermined data and/or commands between the user terminal and 
the targets being connected. 
[0074] 

Then, the page being displayed on the display screen to a projector, or a target, 
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may be transferred as an image to be projected as long as the connection between the 

projector and the user terminal is sustained in said application step (d). 

[0075] 

The user input data input by using said user input means can be transferred as 
user input data on a computer system, or a target, as long as the connection between 
the computer system and the user terminal is sustained in said application step (d). 
[0076] 

Data to be shown can be obtained from the target and displayed on the display 
screen as long as the connection between said targets including said data to be shown 
and updated regularly or irregularly and the user terminal is sustained in application 
step (d). 
[0077] 

Alarms can be received from a target and executed on said user terminal as long 
as the connection between a device having an alarm feature, or a target, and the user 
terminal is sustained in said application step (d). 
[0078] 

The video recording appointment information input at said user terminal to a 
video recording/reproduction apparatus having a video recording appointment feature, 
or a target, can be transmitted to the target so long as the connection between said 
video recording/reproduction apparatus and the user terminal is sustained in said 
application step (d). 
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[0079] 

An image of operation for controlling remotely controllable targets can be 
displayed on the display screen and the user operation displayed on the display screen 
can be transmitted to the targets as long as said the connection between the target and 
the user terminal is sustained in said application step (d), 
[0080] 
[Operation] 

The present invention provides a real world-oriented user interface that can 
sustainedly connect a computer and other information devices in the real space, which 
may be a predetermined space for work. The expression "sustained connection" as 
used herein implies the following characteristic aspects. 
[0081] 

(1) Data can be transferred bi^irectionally. 

(2) Data will not necessarily be transferred (nor need to be transferred) at the moment 
of an operation (when connection is established between devices). 

(3) Devices can be operated interactively so that the transferred data can be controlled 
at the terminal that transmitted the data. 

[0082] 

Such sustained connection is useful particularly when, for example, the user 
uses the note book PC he or she brings into a work space in place of the keyboard of 
the computer installed in the work space and the key arrangement of the installed 
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computer differs from that of the PC the user brings in. Another example may be that 
the user uses a device having a console panel for limited resources such as printer as 
console for maintenance operations. Still another example may be that the user 
transfers transparency data from the computer he or she has at hand to a plurality of 
projectors arranged for presentation in an event and/or controls transparency 
switching operations by means of the computer. 
[0083] 

Known interfaces for displaying information and transferring data may be 
classified into (a) those adapted to identify an object by means of a sensor for 
detecting infrared rays, electric waves or magnetic waves that is held in contact or 
close to the object and (b) those adapted to identify an object by way of visible 
identification information imaged by a camera. 
[0084] 

In the case of interfaces of the type (a), it is highly difficult to sustain the 
connection between the computer and some other object unless the sensor is held in 
contact with the object in view of the fact that a data transfer can occur the instant 
when the connection is established. 
[0085] 

On the other hand, in the case of interfaces of the type (b), the connection 
between the computer and some other object can be sustained by keeping an image 
of the object picked up when the connection is established typically as still image. 
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Additionally, an object that was connected in the past can be selected by retrieving 
and viewing the image picked up when the connection was established just like the 
image of a new object. 
[0086] 

The present invention provides a real world-oriented interface that is adapted 
to establish connection between an object that a computer (e.g., notebook PC) is (or 
was) viewing by way of a camera. This interface has a metaphoric feature that "it 
connects what it sees", which the inventors of the present invention refers to as "gaze- 
link" metaphor. 
[0087] 

FIG. 1 of the accompanying drawings schematically illustrates an operation of 
the feature of "gaze-link" metaphor. Referring to FIG. 1 , as the camera that a portable 
computer is equipped with as one of the standard accessories views the control panel 
of a video recording/reproduction apparatus arranged at the front surface thereof, the 
camera picks up an image of the control panel, which image is then displayed on the 
display screen of the computer to establish connection between the computer and the 
video recording/reproduction apparatus. The established connection is sustained as 
long as the image of the control panel is displayed on the display screen. 
[0088] 

In such a state of connection, data can be transferred bidirectionally and veirious 
operations can be conducted in an interactive way. The video recording/reproduction 
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apparatus is identified by the computer and it is possible to call the functions of the 
control panel on the computer display screen. More specifically, operation buttons 
such as "record", "replay", "stop", "suspend", "fast forward winding" and "fast 
backward winding" are displayed on the display screen and the user can operate the 
video recording/reproduction apparatus by using the buttons displayed on the display 
screen just like he or she uses the proper buttons on the control panel. More over, the 
user can operate the video recording/reproduction apparatus remotely simply by using 
any of the buttons on the control panel. 
[0089] 

The feature of "gaze4ink" metaphor may be taken for as an expansion of the 
metaphoric feature of "retrieving information on what is viewed". The metaphoric 
feature of "retrieving information on what is viewed" is referred to as "magnifying 
glass metaphor" in document [13]. The "magnifying glass metaphor can only retrieve 
fixed information from the object "being viewed". To the contrary, the feature of 
"gaze4ink" metaphor of the present invention can transfer data bidirectionally. 
[0090] 

Taking apart that the "gaze-link" metaphor provides bidirectional data transfer 
and that data can be transferred in a non-fixed way, it can be regarded as a feature of 
repeatedly transferring stand-along small data. On the other hand, document [4] 
proposes a system that can bidirectionally transfer non-fixed information previously 
described. However, the system described in the above document is designed to 
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transfer a stand-along block of data and not adapted to exchange small data for a 

number of times. 

[0091] 

The man's action of "view" has a strong affinity for the concept of connection. 
For instance, in a party where a large number of people enjoying conversation, any 
two who are talking to each other will be looking at the partners as a matter of course. 
Similarly, if you are talking to a person at a remote terminal, you can talk more vividly 
when you can see the partner than when you can just hear what the partner is talking 
so you will probably prefer to see him or her, 
[0092] 

The concept of "seeing" can be expanded from the real world where you can 
see a physical object. For example, if a still image of a viewed object is recorded (and 
stored in the local storage disk of a computer) as described above and then you are 
moved away from the place where you can see the object, you can nevertheless 
establish connection between the object and your terminal by selecting the still image 
(or taking it out from the local storage disk) so that you can "see" the object. 
[0093] 

It is also possible to provide a metaphoric feature that what is "viewed" is a 
recorded video image and the object of the image can be connected by seeing it. 
Referring to FIG. 2 of the accompanying drawings schematically illustrating a scene 
of teleconference, the display screen of the remote site may be connected and shot by 
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a camera by means of the same metaphoric feature. It is also possible to establish 
connection between the main conference room and the remote site to see a stored still 
image. 
[0094] 

The feature of "gaze4ink" metaphor of the present invention can find 
applications that are based on the concept of "connection". Fur example, it is possible 
to switch the telephone call to the user location, or the computer of the user location, 
simply by "viewing" the telephone set (or leaving an image of the telephone set on the 
computer display screen). 
[0095] 

For instance, a system for switching a telephone call to the telephone set at the 
location of the called by using " ActiveBadge"^*"^^ has been developed. However, with 
the feature of "gaze4ink" metaphor of the present invention, the destination of transfer 
of telephone calls can be changed temporarily in a very simple way. Also the user can 
be notified of the reception of a fax letter when the fax machine is "viewed". 
Similarly, the user can be notified of an event such as "water has boiled" or "water has 
ran out" when the electric kettle is "viewed" (assuming that the electric kettle can be 
monitored by way of a telecommunication network. Refer to "Potbiff ' that will be 
des cribed hereinafter) . 
[0096] 

Interactions are allowed as long as connection is sustained. Therefore, it will 
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be possible to realized "multi-purpose remote contror adapted to control objects 
simply by viewing them. Applications for controlling various devices that are being 
"viewed" may also be developed by using the technique of more direct operations of 
GUI (graphical user interface) on video images. 
[0097] 

The above objects and other objects of the present invention as well as the 
advantages of the invention will become clear by reading the following description of 
the preferred embodiments of the invention made by referring to the accompanying 
drawings. 
[0098] 

[Preferred Embodiment of the Invention] 

Now, the present invention will be described in greater detail by referring to the 
accompanying drawings that illustrate preferred embodiments of the invention. 
[0099] 

FIG. 3 is a schematic block diagram of a computer system that can provide the 
feature of "gaze4ink" metaphor, which is a real world-oriented interface according to 
the invention. Referring to FIG. 3, system 100 for the feature of "gaze-link" metaphor 
comprises one or more than one user terminal 10 and one or more than one devices, 
or targets 50), that can be connected to the user terminal. While FIG. 3 shows only 
a single user terminal 10 and a single target 50, there may be a number of user 
terminals and targets in a single word space where network 70 is installed. 
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[0100] 

The real world-oriented interface that operates on the system 100 has a 
metaphoric feature of connecting the user terminal and the target by "viewing" the 
target. In parallel with the realization of the metaphoric feature (or in the background 
thereof), the user terminal 10 and the target 50 are connected to each other by way of 
a LAN (local area network) and/or some other telecommunication network 70 in order 
to exchange data and conmiands. Note, however, it is sufficient for the user terminal 
10 and the target 50 that they can be connected to each other by some means and they 
do not have to be located in close proximity, e.g., in a same room. In other words, 
they may be located in a remote environment where they are located in different 
rooms, floors or buildings and the user (terminal) cannot directly "view" the target. 
[0101] 

For the purpose of the present invention, the network protocol is not limited to 
a specific one. For instance, the LAN may be Ethernet (including both wired and 
wireless) or Bluetooth 
[0102] 

The user terminal 10 represents a computer system that provides a real world- 
oriented interface adapted to connect itself to the target 50 by viewing the latter. The 
present invention is by no means limited to any specific system architectures for 
computers. The user terminal 10 typically is compatible with IBM's PC/ATs 
(personal computer/advanced technology) and their successors in terms of 
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specifications. Note, however, that it should be equipped with a network interface 

card (NIC) or equivalent in order to interconnect itself and the target 50. 

[0103] 

As far as this embodiment is concerned, to "view" an object means that it 
shoots the physical object in the real woiild by means of a camera (and 
recognizes/identifies the imaged object) and to connect by "viewing" means that the 
picked up image of the object is displayed on the display screen of the computer to 
show the connected state. 
[0104] 

To establish connection between the user terminal 10 and the target 50 that is 
"viewed" and hence imaged, the former has to recognize the target 50 to begin with. 
Therefore, for this embodiment, it is assumed that the target 50 includes a visual 
marker such as an ID code that can be visually identified. The ID code may be, for 
example, a bar code in the form of a two-dimensional matrix such as Cybercode/*/. 
The target 50 may be any information device or information-oriented home appliance 
that can be connected to a telecommunication network. For example, it may be a 
video recording/reproduction apparatus, an electric kettle or a projector. 
[0105] 

The user terminal 10 has to be equipped with an imaging means, which is 
typically a camera for "viewing" the object, although the camera may be contained in 
the user terminal 10 (see FIG. 4) or located outside the user terminal 10 (see FIG. 5). 
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A specific example of the user terminal 10 may be "Sony VAIO-CIR", which is a 
notebook type computer comprising a camera as integral part thereof, available from 
Sony Corporation. Such a user terminal 10 can be connected to an outside camera by 
means of a serial (RS-232C) port or an IEEE 1394 (i-Link) port and a cable. 
[0106] 

Furthermore, the user terminal 10 may utilize a camera located on a remote site 
and connected by way of Ethernet (including both wired and wireless) and/or 
BlueTooth (see FIG. 6). Or, the user terminal 10 may not be equipped with a 
camera but can be in a state of "viewing" the target by retrieving a still image of the 
target stored in a local storage disk thereof and displaying it on the display screen. 
[0107] 

As shown in FIG. 3, the user terminal comprises an application section 11, 
connecting section 12 and a data base section 13. 
[0108] 

The application section 1 1 by turn is formed by the following modules. 

[0109] 

(1) A module for recognizing/identifying the ID of the target on the basis of the ID 
code contained in a picked up image of the target 50. 

(2) A module for holding the snap shot of the recognized/identified target 50 obtained 
when it is "viewed" on the display screen selecting it by clicking it by means of a 
pointing device. 
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(3) A module for exchanging data with the selected target or the target that is being 

"viewed". 

[0110] 

The data base section 13 is a module for converting the ID of the target 50 into 
its network address. Upon identifying the ID code of the target 50 in the picked up 
image, the application section 1 1 refers to the data base section 13 for the ID. Then, 
the data base section 13 returns the network address corresponding to the ID. Then, 
the application section 11 specifies the network address to the connecting section 12 
to establish connection between the user terminal 10 and the target 50. 
[0111] 

The entire data base containing the data describing the correspondence of the 
ID of each of the possible targets and its network address may be held under control 
of the user terminal 10. If the data base has to be shared with some other user terminal 
10 or it has a large size due to the fact that a large number of targets exist on the 
network 70, the operation of controlling the data base may be entirely or partly 
assigned to a dedicated data base server 30 on the network 70. Then, the user terminal 
10 inquires of the data base server 30 the ID of the target 50 and the data base server 
30 sends the network address of the target 50 to the user terminal 10. 
[0112] 

The user terminal 10, the target 50, the data base server 30 and other devices 
on the network 70 can exchange messages typically according to the TCP/IP 
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(transmission control protocol/internet protocol). If a large volume of data has to be 
transferred due to a large image size or for some other reason, the feature of sharing 
a file may be utilized. Functional modules such as the application section and the 
server may be described by means of Java Programming Language (an object-oriented 
programming language developed by Sun Microsystems of U. S. A. 
[0113] 

FIG. 7 is an image that may be supplied form the application section 1 1 and 
displayed on a display screen, 
[0114] 

The central area of the display screen is the area to be used by the application 
main body. 
[0115] 

The upper left display area of the display screen shows the image currently 
being picked up by the camera. The lower left display area of the display screen 
shows the image currently being picked up of the connected device. The currently 
connected device is a device being "viewed", which is a device that is being shot by 
the camera or selected by some other means. In the instance of FIG. 7, an image taken 
by the camera of the display monitor that is currently connected is displayed. If the 
connected target has a name or an ID, it may be displayed along the lower edge of the 
window frame. 
[0116] 
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A list of the devices that can be connected is shown at the right end of the 
display screen. The list of the devices to be connected is realized in the form of a 
buttons representing the (still) images of the respective target devices to be connected 
that are taken by a camera and arranged in time series. The connection procedure for 
connecting a desired target and the user terminal 10 can be called by selecting 
(clicking) the button representing the target. 
[0117] 

FIG. 8 is a flow chart of the sequence the processing operation of the 
connecting section 12 for connecting a single target 50. In other words, the user 
terminal 10 can be connected only to a single target. 
[0118] 

Initially, the connecting section is in a standby state due to absence of any 
object to be connected (Step Sll). 
[0119] 

As the camera shoots a target, the application section 1 1 obtains the camera- 
shot image (Step SI 2) and carries out a predetermined processing operation for 
recognizing/identifying the image to determine if the camera-shot image contains an 
ID code such as cyber code. (Step SI 3). 
[0120] 

If the camera-shot image contains an ID code, the application section 1 1 further 
determines if the recognized ID code is same as that of the target being connected or 
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not (Step SI 4). If the two ID codes agree with each other, the operation returns to 
Step S12 to repeat the above steps because the object to be connected does not need 
to be switched. 
[0121] 

If, on the other hand, it is determined in Step S14 that the recognized ID code 
is different from that of the target to be connected, the application section 1 1 further 
determines if the target being connected exists or not (Step SI 5). If the target being 
connected exists, the application section 11 releases the connection (Step SI 6) and 
proceeds to the next step S17. 
[0122] 

In Step S 17, it is determined if the ID code found in the camera^hot image was 
recognized in the past or not. If it was recognized in the past, the image coupled to 
the ID code is retrieved from the local storage disk and displayed on the display screen 
(Step SI 8). If, on the other hand, the ID code has not been recognized, the ID code 
is coupled to the current camera-shot image and added it to the "list of the targets to 
be connected" containing the targets that were viewed in the past (Step SI 9). Then, 
in Step S20, the image coupled to the ID code is displayed on the display screen, 
[0123] 

Then, in Step S21, the application section 1 1 places an inquiry to the data base 
server 30 and obtain the network address of the target corresponding to the ID code. 
Thereafter, in Step S22, connection to the target is established by using the obtained 
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network address. 
[0124] 

It will be appreciated that the user only needs to shoot the target by means of 
a camera in order to establish connection to the target if the feature of "gaze4ink" 
metaphor is installed. Then, the user terminal 10 recognizes the ID code attached to 
the target on the basis of the camera-shot image and automatically reads out the 
network address corresponding to the ID code. 
[0125] 

On the other hand, if it is determined in Step SI 3 that the camera-^hot image 
does not contain any ID code, the operation proceeds to Step S25, where it is 
determined if the user selects an image from the "list of the targets to be connected" 
containing the targets that were viewed in the past or not. 
[0126] 

If it is determined that the user selects an image form the "list of the targets to 
be connected", the operation proceeds to Step S26, where it is determined if there is 
a target that is currently connected to the user terminal or not. If it is determined that 
there if no target that is currently connected to the user terminal, the ID code coupled 
to the selected image is retrieved (Step S27) and then the operation proceeds to Step 
S21 to establish connection to the target shown in the image selected from the "list of 
the targets to be connected" (Step S27). 
[0127] 
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If, on the other hand, it is determined in Step S26 that there is a target currently 
connected tot he user terminal, the operation proceeds to Step S28 to determine if the 
ID code corresponding to the selected image and that of the target being connected 
agree with each other or not. If they agree, the operation returns to Step S 12 to repeat 
the above steps because it is only necessary to maintain the connection. If, on the 
other hand, they do not agree, the current connection to the target is released (Step 
S29) and the operation proceeds to Step S27 to establish connection to the target 
shown in the image selected from the "list of the targets to be connected". 
[0128] 

If the user does not select any image from the "list of the targets to be 
connected" in Step S25, the operation proceeds to Step S30 to determine if there is a 
target that is currently connected to the user terminal or not. 
[0129] 

If there is no target that is currently connected, the operation returns to Step S12 
to repeat the above steps. 
[0130] 

If, on the other hand, there is a target that is currently connected, the operation 
proceeds to Step S31 to determine if the user releases the connection or not. The 
operation returns to Step S12 if the user does not select the release of the connection, 
whereas the operation proceeds to Step S32 to release the connection and returns to 
Step S12 to repeat the above steps if the user selects the releases of the connection. 
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[0131] 

FIG. 9 is a flow chart of the processing operation of the connecting section 12 
that is followed when the user terminal can be connected a plurality of targets 
simultaneously. Now the operation will be described by way of the steps of FIG. 9. 
[0132] 

Initially, the connecting section is in a standby state due to absence of any 
target to be connected (Step S41). 
[0133] 

As the camera shoots a target, the application section 1 1 obtains the camera- 
shot image (Step S42) and carries out a predetermined processing operation for 
recognizing/identifying the image to determine if the camera-shot image contains an 
ID code such as cyber code. (Step S43). 
[0134] 

If the camera-shot image contains an ID code, the application section 1 1 further 
determines if the recognized ID code is same as that of the target being connected or 
not (Step S44). If the two ID codes agree with each other, the operation returns to 
Step S42 to repeat the above steps because the object to be connected does not need 
to be switched. 
[0135] 

If, on the other hand, it is determined in Step S44 that the newly recognized ID 
code is different from that of the object to be connected, the application section 11 
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proceeds to Step S45 where it further determines if the ID code found in the camera- 
shot image was recognized in the past or not. If it was recognized in the past, the 
image coupled to the ID code is retrieved from the local storage disk and displayed 
on the display screen (Step S46). If, on the other hand, the ID code has not been 
recognized, the ID code is coupled to the current camera-shot image and added it to 
the "list of the targets to be connected" containing the targets that were viewed in the 
past (Step S47). Then, in Step S48, the image coupled to the ID code is displayed on 
the display screen. 
[0136] 

Thereafter, in Step S49, the application section 1 1 places an inquiry to the data 
base server 30 and obtain the network address of the target corresponding to the ID 
code. Thereafter, in Step S50, connection to the target is established by using the 
obtained network address. 
[0137] 

It will be appreciated that the user only needs to shoot the target by means of 
a camera in order to establish connection to the target if the feature of "gaze4ink" 
metaphor is installed. Then, the user terminal 10 recognizes the ID code attached to 
the target on the basis of the camera-shot image and automatically reads out the 
network address corresponding to the ID code. 
[0138] 

On the other hand, if it is determined in Step S43 that the camera-shot image 
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does not contain any ID code, the operation proceeds to Step S51, where it is 
determined if the user selects an image from the "list of the targets to be connected" 
containing the targets that were viewed in the past or not. 
[0139] 

If it is determined that the user selects an image form the "list of the targets to 
be connected", the operation proceeds to Step S52, where it is determined if there is 
a target that is currently connected to the user terminal or not. If it is determined that 
there if no target that is currently connected to the user terminal, the ID code coupled 
to the selected image is retrieved (Step S53) and then the operation proceeds to Step 
S49 to establish connection to the target shown in the image selected from the "list of 
the t£irgets to be connected" (Step S53). 
[0140] 

If, on the other hand, it is determined in Step S52 that there is a target currently 
connected tot he user terminal, the operation proceeds to Step S54 to determine if the 
ID code corresponding to the selected image and that of the target being connected 
agree with each other or not. If they agree, the operation returns to Step S42 to repeat 
the above steps because it is only necessary to maintain the connection. If, on the 
other hand, they do not agree, the operation proceeds to Step S53 to establish 
connection to the target shown in the image selected from the "list of the targets to be 
connected". 
[0141] 
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If the user does not select any image from the "Hst of the targets to be 
connected" in Step S51, the operation proceeds to Step S55 to determine if there is a 
target that is currently connected to the user terminal or not. 
[0142] 

If there is no target that is currently connected, the operation returns to Step S42 
to repeat the above steps. 
[0143] 

If, on the other hand, there is a target that is currently connected, the operation 
proceeds to Step S56 to determine if the user releases the connection or not. The 
operation returns to Step S42 if the user does not select the release of the connection, 
whereas the operation proceeds to Step S57 to release the connection and returns to 
Step S42 to repeat the above steps if the user selects the releases of the connection. 
[0144] 

As described above, in Step S21 of the flow chart of FIG. 8 and in Step S49 of 
the flow chart of FIG. 9, the data base section 13 or the data base server 30 is inquired 
of for the ID code that the target has and returns the relevant network address. The 
operation of the inquiry and answer sequence involving the data base server 30 will 
be described below. 
[0145] 

HG. 10 is a flow chart of the processing operation of an "inquiry and answer" 
sequence. In the instance of FIG. 10, if the data base section 13 or the data base server 
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30 does not have any entry of the inquired ID code, it places an inquiry to other data 
base servers that can be located on the network about the ID code in a client/server 
format. Now, the steps of the flow chart will be discussed below. 
[0146] 

Upon receiving an inquiry about an ID code from the connecting section 12 or 
some other data base server (Step S61), the data base server 30 determines if it has a 
local data base of its own or not (Step S62), 
[0147] 

If it possesses a local data base, it refers to it (Step S63) and determines if the 
local data base contains an entry of the input ID code or not (Step S64). 
[0148] 

If the local data base contains an entry, it retrieves the network address of the 
ID code and returns it to the inquirer (Step S65). Thereafter, it returns to Step S61 to 
wait for the next inquiry. 
[0149] 

If, on the other hand, it is determined in Step S62 that the data base server 30 
does not possess any local data base of its own, it determines if there is a data base 
server on the network to which it can place an inquiry or not (Step S66). 
[0150] 

If it finds a data base server on the network, it forwards the inquiry about the 
ID code (Step S67) and receives the network address as answer to the inquiry (Step 
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S68). 
[0151] 

Thus, if the data base server 30 succeeds to obtain the network address from 
some other data base server (Step S69), the operation proceeds to Step S65 to give the 
network address to the inquirer, who may be the connecting section. If, on the other 
hand, the data base server 30 fails to obtain the network address, it notifies the 
inquirer, who may be the connecting section, of the fact that the network address is 
not known (Step S70). 
[0152] 

If it is determined in Step S64 that the local data base does no contain any entry 
of the input ID code, it also determines if there is a data base server on the network to 
which it can place an inquiry or not (Step S66). 
[0153] 

If it finds a data base server on the network, it forwards the inquiry about the 
ID code (Step S67) and receives the network address as answer to the inquiry (Step 
S68). 
[0154] 

Thus, if the data base server 30 succeeds to obtain the network address from 
some other data base server (Step S69), the operation proceeds to Step S65 to give the 
network address to the inquirer, who may be the connecting section. If, on the other 
hand, the data base server 30 fails to obtain the network address, it notifies the 
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inquirer, who may be the connecting section, of the fact that the network address is 

not known (Step S70). 

[0155] 

FIG. 11 is a flow chart of the processing operation of another "inquiry and 
answer" sequence. In the instance of FIG. 1 1, if the data base section 13 or the data 
base server 30 does not have any entry of the ID code about which an inquiry is 
placed, it places an inquiry about the ID code to other data base server that cannot be 
located on the network in a broadcast format. Now, the steps of the flow chart will be 
discussed below. 
[0156] 

Upon receiving an inquiry about an ID code from the connecting section 12 or 
some other data base server (Step S81), the data base server 30 determines if it has a 
local data base of its own or not (Step S82). 
[0157] 

If it possesses a local data base, it refers to it (Step S83) and determines if the 
local data base contains an entry of the input ID code or not (Step S84). 
[0158] 

If the local data base contains an entry, it retrieves the network address of the 
ID code and returns it to the inquirer (Step S85). Thereafter, it returns to Step S81 to 
wait for the next inquiry. 
[0159] 
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If, on the other hand, it is determined in Step S82 that the data base server 30 
does not possess any local data base of its own, it broadcasts the inquiry about the ID 
code onto the network (Step S86). 
[0160] 

If the data base server 30 succeeds to obtain the network address from some 
other data base server as a result of the broadcasting (Step S87), the operation 
proceeds to Step S85 to give the network address to the inquirer, who may be the 
connecting section. If, on the other hand, the data base server 30 fails to obtain the 
network address, it notifies the inquirer, who may be the connecting section, of the 
fact that the network address is not known (Step S88). 
[0161] 

If it is determined in Step S84 that the local data base does no contain any entry 
of the input ID code, it also broadcasts the inquiry about the ID code onto the network 
(Step S86). 
[0162] 

If the data base server 30 succeeds to obtain the network address from some 
other data base server as a result of the broadcasting (Step S87), the operation 
proceeds to Step S85 to give the network address to the inquirer, who may be the 
connecting section. If, on the other hand, the data base server 30 fails to obtain the 
network address, it notifies the inquirer, who may be the connecting section, of the 
fact that the network address is not known (Step S88). 
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[0163] 

Now, an applications that can be provided by a system 100 realizing the feature 
of "gaze-link" metaphor will be discussed below. 
[0164] 

(1) presentation tool 

A number of transparencies stored in the user terminal can be allocated to one 
or more than one targets such as displays and projectors by applying the feature of 
"gaze4ink" metaphor to a presentation tool (see FIG. 12). 

[0165] 

The transparencies transferred to the displays and the proj ectors can be operated 
at the user terminal (e.g., the user can show them one by one or can show them as 
moving picture whose motion can be suspended temporarily). 
[0166] 

FIG. 13 is a flow chart of the processing operation of the feature of "gaze4ink" 
metaphor as applied to a presentation tool. Now, the steps of the flow chart will be 
discussed below. 
[0167] 

The processing operation proceeds with the process of presentation of 
displaying transparencies by means of the user terminal 10. 
[0168] 
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If the user requests some other transparency (Step SlOl), the requested one is 
displayed on the user terminal 10 (Step S102), whereas the current transparency 
remains on the display screen if the user does not request any other transparency. 
[0169] 

Then, in Step S103, the connecting section 12 is inquired if the user terminal 
10 has a target to be connected or not. If the answer to the inquiry is no, the 
processing operation returns to Step SlOl and repeats the above steps. 
[0170] 

If the user terminal 10 has a target to be connected, it is determined if the target 
to be connected is a projector or some other device that can display transparencies or 
not (Step SI 04). If the target is a device that cannot display any transparency, the 
operation returns to Step SlOl and repeats the above steps. 
[0171] 

If, on the other hand, the target is a device that can display transparencies, the 
transparency is transferred to the projector (Step S107) in response to the request of 
the user of the user terminal 10 for transferring the current transparency to the 
projector (Step S106). A request for transferring a transparency may be issued in an 
intuitive way by using, for instance, "hyper-drag" and expanding the current GUI 
operation into the space of the real world. Transparencies are transferred by way of 
the network 70. After the completion of the transfer of data, the processing operation 
returns to Step SlOl and repeats the above steps. 
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[0172] 

If the user requests that the transparency being displayed by the projector that 
is currently connected to the user terminal 10 to be displayed also on the user terminal 
10 (Step SI 08), the projector that is currently connected is inquired about the 
transparency being displayed (Step S109) and the latter is actually displayed on the 
user terminal (Step SI 10). The transfer of data necessary for displaying a 
transparency proceeds automatically by way of the network 70. After displaying the 
transparency, the operation returns to Step S 101 to repeat the above steps. 
[0173] 

If the user does some action in the transparency (Step Sill), which is also 
displayed by the projector that is currently connected (Step SI 12), information on the 
user's action in the transparency is also transferred (Step S 1 1 3) . As a result, the user's 
action is displayed on the projected image of the transparency. Thereafter, the 
processing operation returns to Step S 101 to repeat the above steps. 
[0174] 

(2) keyboard 

If a plurality of computers are located within a limited space and are provided 
with respective keyboards and displays, it may be very annoying to most people to 
operate them properly in such a complex situation. For example, there may be 
occasions where a user keys in a command on a command line on a display screen and 
does not know where the corresponding keyboard is. Then, he or she is forced to do 

62 



a cumbersome operation of ensuring that the connection cable is connected property. 
[0175] 

When a computer is shared by a number of people or a user borrows a 
computer from some other person, the user may experience the difficulty of getting 
accustomed to the key operation particularly when the key arrangement is different 
from that of his or her own keyboard or the keyboard is provided with an input 
program (FEP (front end processor) different from that of the keyboard of his or her 
own. 
[0176] 

However, it is now possible to use the keyboard of a computer that is "viewed" 
and hence connected to the user terminal. 
[0177] 

For example, any application program can be installed into user's own 
computer by using the API (application programming interface) of the window system 
of the computer to which the user terminal is connected. 
[0178] 

Additionally, a metaphoric feature of making the user terminal operate as telnet 
terminal of the computer that is "viewed" as target if it is so arranged that the echo 
back is also displayed. This feature will be very useful for operations on a server 
machine that does not usually need the user of a console or a built4n computer and 
operations for servicing devices having a limited console (e.g., printers). 
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[0179] 

FIG, 14 is a schematic illustration of the feature of "gaze-4ink" metaphor as 
applied to a keyboard. As shown in FIG. 14, as the user "views" a computer (or 
identifies it by way of a camera-shot image thereof), the keyboard of the user terminal 
comes to operate as a remote keyboard of the computer. It is not necessary to connect 
the keyboard of the user terminal to the keyboard port of the remote computer. 
[0180] 

Also as shown in FIG, 14, it may be so arranged that each character entered by 
a key input operation is displayed on the display screen of the user terminal in such 
a way that the user feels as if it were flying to the computer that is "viewed". Such a 
display will be effective for making the user recognize the target connected to the user 
terminal and the state where they are actually connected. 
[0181] 

FIG. 15 is a flow chart of the processing operation of the feature of "gaze4ink" 
metaphor as applied to a keyboard for a remote computer. Now, the steps of the 
operation will be discussed below. 
[0182] 

Initially, the user terminal remains in a standby state in Steps S121 and S122 
until it is connected to a target. 
[0183] 

As the target is "viewed" and the user terminal is connected to the target, it is 
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determined if the target is a device that can receive any key input or not (Step SI 23). 
If the answer is positive, the processing operation returns to Step S121 and the user 
terminal goes back to the standby state until it is connected to some other target. 
[0184] 

If, on the other hand, the user terminal is connected to a device that can receive 
key inputs, e.g., a remote computer, each time the user keys in a character on the user 
terminal (Step SI 24), the input character is transferred to the target connected to the 
user terminal (Step SI 25). 
[0185] 

Then, in Step SI 26, the connecting section 12 is asked if the connected target 
is switched or not. As described above, the connected target can be switched simply 
as the user shifts the object he or she is viewing. 
[0186] 

If it is determined that the connected target is not switched (Step SI 27), the 
processing operation returns to Step SI 24 to repeat the key4n operation at the user 
terminal and the transfer of the input character. 
[0187] 

If, on the other hand, it is determined that the connected target is switched (Step 

51 27) , it is determined if there is a target that is currently connected or not (Step 

5128) . 
[0188] 
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If no target is connected currently, the processing operation returns to Step 
SI 21 and the user terminal is held in a standby state, waiting for the connection of the 
next target. 
[0189] 

If it is determined that there is a target that is currently connected, an operation 
of actually switching the connection of target is carried out (Step SI 29) and the 
processing operation returns to Step SI 24 for transferring the key input of the user 
terminal to the newly connected target. 
[0190] 

(3) bulletin board 

When a large academic conference is held, bulletin boards will typically be set 
up to provide information about the current sessions and the rooms where they are 
going on. If such information is electronically accessible by way of a computer 
network, the attendants of the conference may not have to move around to find a 
bulletin board. However, if the information is accessed only by way of a WWW 
(world wide web) page in order to see the "bulletin board set up at the entrance of the 
building", the access operation will be a painstaking one. 
[0191] 

However, by applying the feature of "gaze4ink" metaphor to a bulletin board, 
the user can see the updated information on the bulletin board simply by "viewing" 
it physically and subsequently retrieving a still image of the bulletin board on the user 
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terminal to establish connection between them. 
[0192] 

Particularly, in the case of a bulletin board showing topics that are periodically 
updated, it may be so arranged that the user can see the contents of particular topics 
sometime later simply by "viewing" the topics on the bulletin board. 
[0193] 

It is also possible to selectively collect items of interest simply by viewing 
them. Then, an internet scrap ^'^^ of news items can be produced. 
[0194] 

FIG. 16 is a flow chart of the processing operation of the feature of "gaze4ink" 
metaphor as applied to BluUetin board. The steps of the operation will be discussed 
below, 
[0195] 

Initially, the user terminal remains in a standby state in Steps S131 and SI 32 
until it is connected to a target, 
[0196] 

As the target is "viewed" and the user terminal is connected to the target, it is 
determined if the target is a device that contains bulletin board data or not (Step 
SI 33). If the answer is negative, the processing operation returns to Step SI 31 and 
the user terminal goes back to the standby state until it is connected to some other 
target. 
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[0197] 

If, on the other hand, the user terminal is connected to a bulletin board, the user 
terminal request the updated bulletin board data (Step SI 34). Upon receiving the 
bulletin board data from the connected target, the user terminal displays them on its 
own display screen (Step SI 35). 
[0198] 

Then, in Step SI 36, the connecting section 12 is asked if there is any other 
target to be connected or not. 
[0199] 

If there is no target to be connected (Step SI 37), the processing operation 
returns to Step S131 and the user terminal is held in a standby state, waiting for the 
connection of the next target. 
[0200] 

If there is some other target to be connected (Step SI 37), it is determined if the 
target to be connected is a device containing bulletin board data or not (Step SI 38), 
If the target is a device containing bulletin board data, the data are obtained and 
displayed on the display screen. If, on the other hand, the target is not a device 
containing bulletin board data, the processing operation returns to Step S131 and the 
user terminal is held in a standby state, waiting for the connection of the next target. 
[0201] 

(4) other applications of the feature of "gaze4ink" metaphor 
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With the feature of "gaze-link" metaphor, it is possible to realize a real world- 
oriented interface adapted to operate with a portable computer equipped with a camera 
and establish connection between the user terminal and the object being viewed by the 
camera. 
[0202] 

For instance, the feature of "gaze4ink" metaphor may be used in an office or 
a residence to connect the user terminal to various information-oriented devices 
including electric home appliances and maintain the connection. Then, an OA (office 
automation) device such as a fax machine or an electric home appliance such as a 
refrigerator or an electric kettle can be connected to the user terminal simply by 
"viewing" it so that changes, if any, in the condition of the device can be monitored 
and the device can be operated by issuing commands to it. 
[0203] 

An example of application of the feature of "gaze-link" metaphor to an 
information-related electric home appliance is Potbiff . 
[0204] 

FIG. 17 is a schematic illustration of the operation of the feature of "gaze^ink" 
metaphor as applied to an information-oriented electric kettle, or "Potbiff. 
[0205] 

Referring to FIG. 17, connection is established between the user terminal and 
the electric kettle as a still image of the electric kettle "viewed" by the user is 
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displayed on the display screen of the user terminal. Additionally, their connection 
is sustained as long as the still image is displayed on the display screen. Under this 
condition, the user terminal can be notified of the fact that water is boiling in the kettle 
and monitor the hot water remaining in the kettle. 
[0206] 

In an office where an electric kettle is provided for making coffee and tea, it 
may not be visible to everybody in the office. Then, a person who want a cup of tea 
may walk to the kettle only to find that water is not boiling in the kettle or the kettle 
is almost empty. However, with Potbiff, it is now possible to establish connection 
between the user terminal and the electric kettle by "viewing" the kettle so that the 
user may be notified of the fact that water is boiling in the kettle or remotely monitor 
how much hot water is left in the kettle from his or her own chair. 
[0207] 

It may be needless to say that the feature of "gaze4ink" metaphor can be 
introduced to not only an electric kettle but also any other information-oriented 
electric home appliance. For instance, the end of a washing cycle of an electric 
laundry machine or that of a heating operation of a microwave cooker can be remotely 
detected without raising the volume of the alarm. Therefore, Potbiff will be 
particularly useful for coin-driven laundry machines. 
[0208] 

FIG. 18 is a flow chart of the processing operation of the feature of "gaze^ink" 

70 



metaphor as applied to an electric kettle for carrying out Potbiff . Now the steps of the 

flow chart will be discussed below. 

[0209] 

Initially, the user terminal remains in a standby state in Steps S141 and SI 42 
until it is connected to a target. 
[0210] 

As the target is "viewed" and the user terminal is connected to the target, it is 
determined if the target is a device that issues an alarm (e.g., an electric kettle that 
sounds an alarm when water boils in it) or not (Step S143). If the answer is negative, 
the processing operation returns to Step S141 and the user terminal goes back to the 
standby state until it is connected to some other target. 
[0211] 

If, on the other hand, the user terminal is connected to a device that issues an 
alarm, the user terminal maintains the connection (Step S144) and monitors the 
device. Upon receiving an alarm from the connected target (Step S145), the user 
terminal displays the message of the alarm (e.g., "water is boiling) on its own display 
screen (Step SI 46), 
[0212] 

Another example of application of the feature of "gaze4ink" metaphor to an 
information-oriented electric home appliance is a video recording appointment system. 
[0213] 
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Recently, there have been marketed systems adapt to make an appointment of 
recording a television program by using a PC provided with a feature of television 
tuner and that of hard disk recording and simply by clicking the program on the table 
of television programs on WWW Such a system may be combined v^ith a video 
recording/reproduction apparatus that can be controlled by way of a serial cable (e.g., 
i Link) so that it is possible to make an appointment of recording a television program 
by clicking the "anchor" of the television program that the user wants to see on the 
WWW page displayed on the display screen and "viewing" the video 
recording/reproduction apparatus. 
[0214] 

FIG. 19 is a flow chart of the processing operation of the feature of "gaze-link" 
metaphor as applied to a video recording appointment system. Assume that the user 
terminal and the video recording/reproduction apparatus are connected by way of a 
network such as BlueTooth or a serial cable such as RS (recommended standard)- 
232C or ULink so that data and commands may be transmitted therebetween. Now, 
the steps of the flow chart will be discussed below. 
[0215] 

Initially, the user terminal remains in a standby state in Steps S151 and S152 
until it is connected to a target. 
[0216] 

As the target is "viewed" and the user terminal is connected to the target, it is 
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determined if the target is a video recording/reproduction apparatus or not (Step 
S153). If the answer is negative, the processing operation returns to Step S151 and 
the user terminal goes back to the standby state until it is connected to some other 
target. 
[0217] 

If, on the other hand, the user terminal is connected to a video 
recording/reproduction apparatus, it is determined if there is a user input relating to 
information on an appointment of recording a video program on the user terminal or 
not (Step 8154). If it is determined that there is a user input, the user terminal 
transmits the information on the appointment to the video recording/reproduction 
apparatus to which it is connected (Step SI 55). 
[0218] 

Then, in Step SI 56, the connecting section 12 is asked if the connected target 
is switched or not. As pointed out above, the connected device can be switched 
simply as the user shifts the object he or she is viewing. 
[0219] 

If it is determined that the connected target is not switched (Step S157), the 
processing operation returns to Step SI 54 to repeat the operation of receiving a user 
input relating to information on an appointment of recording a video program on the 
user terminal and transmitting the information to the video recording/reproduction 
apparatus. 
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[0220] 

If, on the other hand, it is determined that the connected target is switched (Step 

5157) , it is determined if there is a target that is currently connected or not (Step 

5158) . 
[0221] 

If no target is connected currently, the processing operation returns to Step 
S15 1 and the user terminal is held in a standby state, waiting for the connection of the 
next target. 
[0222] 

If it is determined that there is a target that is currently connected, an operation 
of actually switching the connection of target is carried out (Step SI 59) and the 
processing operation returns to Step SI 54 to repeat the operation of receiving a user 
input relating to information on an appointment of recording a video program on the 
user terminal and transmitting the information to the video recording/reproduction 
apparatus. 
[0223] 

Another example of application of the feature of "gaze4ink" metaphor is 
"multi-purpose remote control" or "remote maintenance" by means of which what is 
"viewed" can be controlled. 
[0224] 

With "multipurpose remote control", the user terminal can be connected to any 
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target simply by "viewing" the target and control the operation of the target. It is not 
necessary that the target is located in close proximity for "viewing" it in order to 
establish connection between the user terminal and the target. In stead, the target can 
be connected to the user terminal by "viewing" it remotely. Nor is it necessary to 
"view" the target directly. In stead, the target can be connected by retrieving a still 
image obtained in the past by "viewing" it in order to realize "remote maintenance". 
[0225] 

FIG. 20 is a flow chart of the processing operation of the feature of "gaze-link" 
metaphor as applied to a remote maintenance system. Now, the steps of the flow chart 
will be discussed below. 
[0226] 

Initially, the user terminal remains in a standby state in Steps S161 and SI 62 
until it is connected to a target. 
[0227] 

As the target is "viewed" and the user terminal is connected to the target, it is 
determined if the target is a device that can be remotely controlled or not (Step SI 63). 
If the answer is negative, the processing operation returns to Step S161 and the user 
terminal goes back to the standby state until it is connected to some other target. 
[0228] 

If, on the other hand, the user terminal is connected to a remotely controllable 
target, an image for controlling/operating the target is displayed on the display screen 
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of the user terminal (Step SI 64). If the remotely controllable target is a video 
recording/reproduction apparatus, a GUI image including function buttons for replay, 
recording, fast forward winding, fast rewinding, pause and other functions is displayed 
on the display screen. Alternatively, an image obtained by shooting the front control 
panel of the video recording/reproduction apparatus and incorporating function 
buttons therein in the form of icons may be used as a GUI image. With the latter 
arrangement, the user can remotely control the operation of the video 
recording/reproduction apparatus as if he or she were remotely controlling a real video 
recording/reproduction apparatus. 
[0229] 

As the user operate one or more than one function keys on the display screen 
of the user terminal as user input (Step S 165), the control information corresponding 
to the operated function keys is transferred to the connected target (Step SI 66). 
[0230] 

Then, in Step SI 67, the connecting section 12 is asked if the connected target 
is switched or not. As pointed out above, the connected device can be switched 
simply as the user shifts the object he or she is viewing. 
[0231] 

If it is determined that the connected target is not switched (Step SI 68), the 
processing operation returns to Step S165 to repeat the operation of receiving a user 
input of operating one or more than one function keys on the display screen of the user 
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terminal and transferring the control information corresponding to the operated 

function keys to the target. 

[0232] 

If, on the other hand, it is determined that the connected target is switched (Step 
SI 68), it is determined if there is a target that is currently connected or not (Step 
SI 69). 
[0233] 

If no target is connected currently, the processing operation returns to Step 
S161 and the user terminal is held in a standby state, waiting for the connection of the 
next target. 
[0234] 

If it is determined that there is a target that is currently connected, an operation 
of actually switching the connection of target is carried out (Step SI 70) and the 
processing operation returns to Step SI 65 to repeat the operation of receiving a user 
input of operating one or more than one function keys on the display screen of the user 
terminal and transferring the control information corresponding to the operated 
function keys to the target. 
[0235] 

If an electric home appliance becomes malfunctioning, it is possible to transfer 
information relating to the malfunction by connecting the user terminal to a support 
center of the manufacturer or distributor simply by viewing the home appliance from 
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behind the camera of the user terminal. It is also possible to eliminate the cause of the 

malfunction by modifying the specified values of some of the components, utilizing 

the remote maintenance function. 

[0236] 

«N.B.» 

* Cybercode is a type of two-dimensional bar code and comprises "a guide bar 
region" for showing the location of the cybercode and a "code pattern region" for 
showing a two-dimensional code pattern as shown in FIG. 24. The code pattern 
region contains cells arranged to form an n x m matrix (7 x 7 in FIG. 24), each cell 
being either a white cell or a black cell to form a binary system. Note that black cells 
are always arranged at the four corners of the code pattern region to make them 
operate not as information carriers but as registration cells. Cybercode can be used 
to identify a total of 1}^ objects by changing the arrangement of the printed cells. 

The procedure of recognizing Cybercode comprises a step of binarizing the 
picked up image thereof, a step of finding guide bar candidates from the binarized 
image, a step of detecting the corner cells on the basis of the position and the direction 
of the guide bar and a step of decoding the bit map pattern of the image in response 
to the detection of the guide bar and the corner cells. 

Then, the ID information and the positional information contained in the code 
can be recovered by carrying out an error bit check operation to make sure that the 
picked up image contains a correct cyber code. Any distortions that may be generated 
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in the code as a result of inclination of the camera and/or the object can be 
compensated on the basis of the positions of the corner cells. 

For more information on Cyber code, refer to, inter alia, Japanese Patent 
Application No. 10-1 84350 assigned to the applicant of the present patent application 
("Image Processing Apparatus and Method as well as Media for Providing Images"). 
[0237] 
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[0238] 

[Supplement] 

The present invention is described in detail by way of specific embodiments. 
However, it is obvious that those who are skilled in the art can modify or alter the 
embodiments without departing from the scope of the invention. Therefore, the above 
described embodiments are simply non4iniitative examples and the scope of the 
present invention is limited only by the claims that follow. 
[0239] 

[Effect of the Invention] 

Therefore, according to the present invention, it is possible to provide a 
comfortable user interface environment where users can operate various devices 
arranged in an information space in an easy, intuitive and integrated way by means of 
the user computer. 
[0240] 

81 



According to the present invention, it is also possible to provide a comfortable 
user interface environment where users can control and operate various devices 
arranged in an information space in an easy and intuitive way on the user computer 
display. 
[0241] 

According to the invention, it is also possible to provide a comfortable user 
interface environment where users can sustainedly connect various devices arranged 
in an information space to computers. 
[0242] 

According to the invention, it is also possible to provide a comfortable user 
interface environment where users can sustainedly connect various devices arranged 
in an information space to computers and confirm the connected state on the user 
computer display. 
[0243] 

According to the invention, it is also possible to provide a comfortable user 
interface environment where users can sustainedly connect various devices arranged 
in an information space to computers and cause them to exchange commands and data. 
[0244] 

According to the invention, there is provided a feature referred to as "gaze-link" 
metaphor that can be used to establish connection between a portable computer 
equipped with a camera and an object that is viewed by the camera. For example, if 
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ID codes such as "cyber codes" (previously described) are assigned and applied to the 
respective objects to be connected to the portable computer, the computer will 
recognize any of the objects on the basis of the image thereof shot by the camera and 
establish connection between itself and the selected object. 
[0245] 

According to the invention, it is now possible to introduce the concept of 
sustained "connection" as opposed the conventional techniques of sequential transfer 
of data, using known real world-oriented interfaces. As a result, the user can 
intuitively and directly specify an object to be operated by using the feature of "gaze- 
link" in a telecommunication network environment where a number of devices are 
mutually connected in an office or home without being worried about the IDs and the 
network addresses of the devices. In other words, users are relieved of the 
cumbersome work of assigning names to the devices and memorizing them. 
[0246] 

Furthermore, it is not necessary to "view" an object directly. In stead, a remote 
object may be viewed by way of a video image or a photograph image to operate real 
world-oriented interfaces. Furthermore, images taken by the camera may be stored 
in the computer so that an image that has been "viewed" to connect the object of the 
image may be selected from the storage and reused for connection. Additionally, 
since images of objects are used in place of the objects themselves, an object can be 
connected to a remote terminal by way of an image taken by a video camera in a 
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teleconferencing environment. 
[Brief Description of the Drawings] 
[HG. 1] 

Schematic illustration of an operation using the feature of "gaze4ink" metaphor 
according to the invention; 
[FIG. 2] 

Schematic illustration of a metaphoric feature with which an image is "viewed" 
indirectly and an object shown in the image is connected and, more specifically, it 
shows a scene of teleconference, where the display screen of a remote site is 
connected and shot by a camera by means of the metaphoric feature; 
[FIG. 3] 

Schematic block diagram of a computer system that can provide the feature of 
"gaze-link" metaphor, which is a real world-oriented interface according to the 
invention; 
[FIG. 4] 

Schematic illustration of a user terminal 10 equipped with a camera or a user 
terminal 10 comprising a camera as an integral part thereof; 
[FIG. 5] 

Schematic illustration of a user terminal 10 equipped with a camera or a user 
terminal 10 to which a camera is connected by way of a cable; 
[HG. 6] 
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Schematic illustration of a user terminal 10 equipped with a camera or a user 
terminal 10 to which a remote camera is connected by way of a telecommunication 
network and/or telephone lines; 
[FIG. 7] 

Image that is supplied form the application section 1 1 and displayed on a 
display screen; 
[FIG. 8] 

Flow chart of the processing operation of the connecting section 12 for 
connecting a single target 50; 
[FIG. 9] 

Flow chart of the processing operation of the connecting section 12 for 
connecting a plurality of targets simultaneously; 
[FIG. 10] 

Row chart of the processing operation of an "inquiry and answer" sequence; 
[FIG. 11] 

Flow chart of the processing operation of another "inquiry and answer" 
sequence; 
[FIG. 12] 

Schematic illustration of the feature of "gaze4ink" metaphor as applied to a 
presentation tool, where the transparencies stored in the user terminal are allocated to 
more than one targets including one or more than one displays and/or one or one or 
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more than one projectors; 
[HG. 13] 

Flow chart of the processing operation of the feature of "gaze-4ink" metaphor 
as applied to a presentation tool; 
[FIG. 14] 

Schematic illustration of the feature of "gaze4ink" metaphor as applied to a 
keyboard, where the keyboard of the user terminal operates as that of the computer 
that the user "viewed"; 
[FIG. 15] 

Flow chart of the processing operation of the feature of "gaze-4ink" metaphor 
as applied to a keyboard for a remote computer; 
[HG. 16] 

Flow chart of the processing operation of the feature of "gaze-link" metaphor 
as applied to BluUetinboard; 
[FIG. 17] 

Schematic illustration of the feature of "gaze4ink" metaphor as applied to an 
information-oriented electric home appliance, which is an electric kettle, in order to 
monitor and notify a state where water is boiling in the electric kettle or no hot water 
is left in it on the display screen of the user terminal; 
[HG. 18] 

Flow chart of the processing operation of the feature of "gaze-link" metaphor 
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as applied to an electric kettle for carrying out Potbiff; 
[FIG. 19] 

Flow chart of the processing operation of the feature of "gaze-4ink" metaphor 
as applied to a video recording appointment system; 
[FIG. 20] 

Flow chart of the processing operation of the feature of "gaze^ink" metaphor 
as applied to a remote maintenance system; 
[FIG. 21] 

Schematic illustration of a Pick-and-Drop operation and, more specifically, it 
shows how a GUI operation is succeeded from the display screen of a notebook type 
computer to that of another computer and a digital object on the former display screen 
is dropped on the other display screen that are separated from each other in the real 
space; 
[FIG. 22] 

Schematic illustration of a Hyperdrag operation and, more specifically, it shows 
how the cursor getting to an edge of the display screen of a computer as a result of the 
operation of the mouse is moved onto the table carrying the computer; 
[FIG. 23] 

Schematic illustration of another Hyperdrag operation and, more specifically, 
it shows how a dig object that has jumped out from the display screen of a computer 
onto the table carrying the computer is further "hyperdragged" to move onto a wall; 
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[FIG. 24] 

Schematic illustration of a cyber code. 
[Explanation of Referenced Numerals] 

10 user terminal 

1 1 application section 

12 connecting section 

13 data base section 
30 data base server 
50 targets 

70 network 
100 system 
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[Name of Document] ABSTRACT 

[Summary] 

[Task] 

Each device arranged in an information space and a computer are sustainedly 
connected and the connection can be visually recognized on the display screen of the 
computer. 

[Means for Solution] 

More specifically, there is provided a feature of "gaze-link" metaphor with 
which connection between the computer and an object viewed by a camera is 
established. A visible marker may be applied to the object of connection so that the 
object may be identified by shooting the object by means of a camera and connection 
may be established between the computer and the object. As the connection between 
the identified object and the computer is sustained, the user can specify the target of 
operation in an intuitive and direct manner without worrying about the ID of each 
device in a network environment where a large number of devices are mutually 
connected. 

[Selected Drawing] Fig. 1 
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